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ABSTRACT 

A preliminary investigation has been made of some 
claims regarding alternative assessment. Using the California 
Learning Assessment System (CLAS) — Middle Grades Mathematics 
Performance Assessment as a platform, the study examined how 
alternative assessment operates in actual practice. Early findings 
are presented in the following areas: (1) students' approaches to 
novel open-ended tasks as compared to familiar multiple-choice tasks; 
(2) student attitudes toward such tasks as compared to familiar 
multiple-choice tasks; and (3) students 1 opportunity to learn on the 
new assessments across cultural background and socioeconomic status. 
The study sample involved 13 schools with 27 teachers (about 67% from 
urban schools) . Interviews, observations, student surveys, reviews of 
instructional materials, and student archival data were used. 
Students appear to understand the approach required by alternative 
assessment and find the problems interesting and challenging. They 
tend to express a preference for multiple-choice items. Suburban and 
nonminority students appear to feel better prepared and to have 
better access to recent texts. An appendix contains the study 
instruments. Thirty-nine tables present study findings. (Contains 12 
references . ) (SLD) 
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A FIRST LOOK: 

ARE CLAIMS FOR ALTERNATIVE ASSESSMENT HOLDING UP? 

Joan L. Herman 
DavinaC. D.Klein 
Tamela M. Heath 
SaraT.Wakai 

INTRODUCTION 

Educational policy makers at the national, state, and local levels continue 
to act on their beliefs in the power of educational assessment to improve 
schools. Through new mandated assessments, policy makers believe they can 
communicate standards; motivate and monitor progress toward attainment of 
those standards; provide useful feedback to all in the school community; and 
hold schools, and the teachers and students within them, accountable for 
improved performance. Their beliefs are bolstered by research showing that 
traditional testing has encouraged teachers and students to focus on what is 
tested (Herman & Golan, 1991; Madaus, 1991; Shepard, 1991). Unfortunately, 
due to the test content on traditional standardized tests, this teaching-to-the- 
test has resulted in a distortion of the curriculum for many students, 
narrowing it to basic, low-level skills (Dorr-Bremme & Herman, 1983; Herman 
& Golan, 1991; Kellaghan & Madaus, 1991; Shepard, 1991; Smith & Rottenberg, 
1991). The result: Teachers, administrators, and policy makers across the 
country are seeking new kinds of assessments whose content will reflect 
rigorous standards for student accomplishment; thus, these new assessments 
will encourage schools to teach and students to learn the complex knowledge 
and problem-solving skills needed for future success. 

Unlike traditional tests, new alternative assessments encourage students 
to think critically and draw their own conclusions to complex problems. 
Rather than asking students to select answers to short, discrete questions — 
often devoi d of real-world context or application — these new assessments invite 
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students to create extended responses, using multiple modes of representation. 
New assessments minimize the importance of rigid time constraints; they also 
encourage students to use tools (such as calculators) to help them in solving 
the novel problems on the assessment. Students' responses to real-life, 
"authentic" problems are scored by educators exercising judgment, not by 
machines reading "bubbles;" students' thinking processes, as well as their 
products, are often taken into consideration in the scoring rubrics. 

Claims for these new kinds of assessments are frequent in the literature 
and on national conference agendas. One claim is that these assessments 
truly stimulate students to engage in complex thinking and thus reflect higher 
standards of excellence than old-style standardized tests. Their ability to target 
higher level thinking and problem-solving skills makes these assessments 
suitable targets for instruction. Of course, a critical link in the policy chain is 
the necessity of having teachers who are prepared to help students develop the 
complex knowledge and skills that these assessments aim to teach. Another 
claim is that students will find these assessments more meaningful and 
motivating than traditional tests. These claims stem from the realistic and 
complex nature of the problems: All students should be encouraged by these 
types of tasks to show what they know and can do, rather than just those 
students who are motivated by the external rewards afforded them in high 
standardized test scores. Finally, a third claim has been that perhaps these 
new types of assessments will help close the equity gap seen on traditional 
tests. 

While the rhetoric is abundant, due to the relative newness of alternative 
assessments evidence substantiating the above claims is just beginning to be 
accumulated. Furthermore, national and state dialogues have been relatively 
silent about the potential equity issues posed by these new assessments. Some 
researchers, however, have begun to articulate significant equity issues which 
need to be investigated for new assessments. For example, the CRESST 
criteria for judging the quality and validity of an assessment highlight 
fairness as a major concern (Herman, 1992; Linn, Baker, & Dunbar, 1991). 
Does an assessment equitably consider the cultural background of all students 
taking the test? Beyond traditional concerns for stereotyping, bias, and 
differential item functioning, does an assessment enable all students to 
demonstrate their real progress and capability? Is it motivating for all 
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students? Winfield and Woodard (1992, in press) warn that alternative 
measures are at least as likely as traditional measures to disadvantage 
students of color; they fear that because there will be time to administer only a 
relatively small number of alternative assessment tasks, the probability 
increases that those tasks will unfairly represent tasks more familiar and 
meaningful to the dominant culture. 

Linn et al. (1991) point to additional fairness concerns stemming from 
students' opportunity to learn that v/hich is assessed on alternative 
assessments. Opportunity to learn (OTL) is defined as the instructional 
opportunities and access to resources that would enable students to develop the 
complex thinking and problem-solving skills that are the targets of the new 
assessments. Since research suggests that disadvantaged students have been 
the most negatively affected by the traditional test-driven curriculum, these 
students have probably had the least opportunity to develop complex thinking 
skills and deep understanding (Herman & Golan, 1993). Further, there are 
concerns that some of the resources and tools that are critical to new ways of 
teaching and learning — calculators, scientific manipulatives, other 
instructional artifacts — will be differentially available to schools serving 
different communities. As a result, some fear that the differences between 
high and low socioeconomic status (SES) students will be even more dramatic 
on alternative assessments than on traditional ones. 

Overview of the CLAS Study 

The study reported here is a preliminary investigation of some of the 
claims made regarding alternative assessment. Using the California 
Learning Assessment System (CLAS) Middle Grades Mathematics 
Performance Assessment as a platform, the study examined how alternative 
assessment operates in actual practice. Do the claims discussed above hold 
up? In this paper, we present early findings in three areas: 

• Students' approaches to novel open-ended tasks as compared to 
familiar multiple-choice tasks; 

• Students' attitudes towards novel open-ended tasks as compared to 
familiar multiple-choice tasks; and 
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• Students' opportunity to learn the skills and tasks on the new 
assessment, compared across students of different cultural 
backgrounds and SES levels. 

At the forefront of state efforts to design new approaches to assessment, 
the CLAS provides a good opportunity to investigate such issues. The 
assessment program features a matrix sampling design, and at the middle 
grades (eighth-grade level) uses a total of eight mathematics assessment 
forms. Each assessment form consists of two sections, the first containing two 
open-ended tasks and the second composed of eight multiple-choice items. The 
two open-ended tasks are designed to pose aathentic, relevant problem 
situations for students to solve; the multiple-choice items are intended to 
asse.- mathematical thinking. In the future, the CLAS plans gradually to 
phase in other assessment types, such as multiday mathematical 
investigations and portfolios. 

METHODOLOGY 



Design of the CLAS Study 

The study's original design sought to contrast schools across the state 
serving diverse school communities. Because of equity concerns, the contrasts 
of particular interest were between schools serving relatively affluent 
suburban communities and schools thought to be potentially at risk— those 
serving inner-city, economically disadvantaged communities and those in 
more geographically remote rural areas. In addition, because inner-city 
students were considered most at risk, and because the cultural implications 
of authentic tasks seemed deserving of inquiry, we deliberately planned to 
overrepresent inner-city schools and to have majority representation of various 
cultural minorities (African-American, Latino, and Asian-American). 
Within each school, we planned to randomly select three eighth-grade math 
classes for study. These classes were to represent the range of eighth-grade 
classes typically taught at that school. 

Our study design was based on a larger pilot study conducted by the state 
and largely dependent on their volunteer sample. As a result, our initial 
design specifications were not fully realized. The final sample consisted of 13 
schools across the state, distributed over three broad categories of schools: 
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urban, rural, and suburban. It encompassed 27 teachers (including 66.7% 
from urban schools, 14.8% from rural schools, and 18.5% from suburban 
schools) and over 800 students (including 56.8% from urban schools, 20.6% 
from rural schools, and 22,6% from suburban schools). (See Table 1 for 
breakdown of school sample.) 

The urban schools were all economically disadvantaged and reflected a 
range of ethnic diversity — principally Latino; mixed Afirican-American and 
Latino; mixed Asian-American and White; mixed White, African-American, 
and Latino. The suburban schools served predominantly White and some 
Asian- American high wealth communities. The rural schools were mixed in 
socioeconomic status and served mainly White and Latino students. See 
Table 2 for an ethnic breakdown of the students in our sample and Table 3 for 
breakdown by socioeconomic status indices, based on demographic data 
supplied by the state. 

Table 1 



Breakdown of Schools Participating in the CLAS Study 




Number of 


Number of 


Type of school 


schools 


classes 


Urban 


9 


24 


Rural 


2 


6 


Suburban 


2 


6 



Table 2 

Percentage of Students of Each Ethnicity by School Type 



Type of 


African- 


Asian- 






school 


American 


American 


Latino 


White 


Urban 


32.8 


20.3 


25.1 


20.5 


Rural 






33.3 


66.7 


Suburban 


1.9 


25.7 


3.8 


68.1 



Note. Percentages may not sum to 100% due to small Native American 
populations. 
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Table 3 

Socioeconomic Status of Participating Schools: Percentage of Students Reporting Various 
Parents' Education Levels by School Type 



Type of 
school 


Not high school 
graduate 


High school 
graduate 


Some 
college 


College 
graduate 


Has advanced 
degree 


Urban 


22.9 


21.0 


24.8 


23.5- 


7.8 


Rural 


6.3 


18.8 


43.8 


25.0 


6.3 


Suburban 


1.3 


1.9 


16.3 


39.4 


41.3 



Instrumentation 

The study utilized six different data sources: (a) classroom observations of 
CLAS administrations (focusing on special, pilot study CLAS administrations 
where all students responded to a common form); (b) student surveys; 
(c) student retrospective interviews; (d) teacher interviews; (e) collection of 
instructional materials, including samples of classroom assignments and 
tests; and (f) archival data on student grades, attendance records, and 
standardized test scores. This paper includes data from the first four sources 
only. 

Classroom observations. Special, pilot-level CLAS administrations were 
observed by two researchers. Using a standard protocol, observers collected 
information on administration conditions, students' reactions to the 
assessment, their engagement level, their use of calculators, and how much 
time students spent on the assessment (see Appendix for copies of all the 
instruments). 

Student surveys. All students in sampled classrooms completed a survey 
on the day following the administration of the pilot-level CLAS. The survey 
solicited students' views on a number of issues, including: students' 
instructional experience with and preparation for the specific knowledge, 
skills, and task types encountered on the CLAS; their access to calculators at 
home and at school; their attitudes towards math in general; and their 
affective responses to open-ended tasks compared to multiple-choice tasks. 

Retrospective student interviews. In-depth student interviews were 
conducted with six students randomly chosen from each classroom. The 
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individual student interviews allowed researchers to obtain more detailed 
information on student responses to the open-ended and multiple-choice tasks 
included in the assessment. Think-aloud protocols asked students to recreate 
their thinking processes and expectations as they approached and tried to solve 
one of the two open-ended tasks and the first multiple-choice item included on 
the common form. The interviews also asked students to explain how they 
thought each task would be scored, their level of preparation for specific itemc., 
and their relative preferences, along a number of affective dimensions, for 
open-ended versus multiple-choice problems. 

Teacher interviews. Finally, teacher interviews were conducted to obtain 
information about teachers' educational background and teaching experience, 
particularly in mathematics; their pedagogical practices; their familiarity 
with and the extent to which they prepared their students for CLAS-type items; 
calculator instruction and use in their classrooms; and their reactions to the 
CLAS. In addition, during the interview, teachers were asked to provide 
researchers with (a) descriptions of major assignments given to students 
during the year and (b) samples of tests and quizzes given during the year. 

Student data. Subsequent to on-site data collection, schools were asked to 
provide data on individual-level student grades, attendance, and standardized 
test scores. While not available for this current report, the study also will have 
access to state data, including actual student performance on the assessment 
and individual demographic and special program participation information 
solicited via the Student Information Form. 

Data Coding and Reliabilities 

Categories for coding responses to open-ended questions on our 
instruments were derived by reviewing a sample of responses; the major 
themes and/or key ideas so-identified were then operationally defined and used 
to categorize each response. 

Interrater reliability was established by double coding a set proportion of 
responses, with the proportion varying depending on the complexity of the 
coding categories. Because of their complexity, all student retrospective 
interview responses dealing with how students approached individual 
assessment tasks were coded by two raters; interrater agreement on these 
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questions ranged from 0.69 to 1.0, with a median of 0.91. Only a sample (25%) 
of the student interview open-ended attitude responses was double-coded since 
coding categories for these were more straightforward. Interrater agreements 
on these items confirmed this judgment, as rater agreements ranged from 
0.85 to 1.0, with a median of 0.94. 

Similarly, 25% of the open-ended responses on the classroom observation 
forms and teacher interviews were double-coded. Interrater agreements from 
these coding schemes were similarly high; correlation coefficients ranged 
from 0.78 to 1.0, with a median of 0.93. 



Data Analyses 

Data were coded at the individual student and classroom levels, where 
possible. Analyses then were conducted at either the individual or classroom 
levels, depending on the specific variable of interest. Individual-level analyses 
were used to explore students' approaches to the assessment and students 
attitudes— areas where individual differences were likely to be predominant. 
Student responses to individual items also were aggregated into two scales, 
based on factor analysis results. One scale denoted students' attitudes toward 
open-ended questions; the other related to their attitudes toward mathematics. 
Both chi square and analysis of variance techniques were used to explore 
differences in individual-level responses by school type. 

Our analyses of differences in classroom practices, however, utilized the 
classroom rather than the individual student as the unit of analysis. Thus, 
data collected from students that explicitly measured classroom-level 
opportunity to learn (OTL) variables were aggregated as classroom means on 
these specific measures, and then analysis of variance was used examine 
differences in classroom experiences by school type (i.e., urban, rural, and 
suburban). Note that the relatively small sample of teachers and classrooms 
represents a significant constraint on the power of our analyses and an 
important caveat in interpreting some of the results that follow. 

In the sections below, we present preliminary results on claims made 
regarding alternative assessment in three areas: students' approaches to 
novel open-ended and familiar multiple-choice tasks, students' attitudes 
toward these two types of tasks and toward mathematics in general, and 
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general issues in students' opportunity to learn. Note that additional analyses 
are continuing, examining in greater detail equity in opportunity to learn and 
drawing on state data of student performance and demographic data. 

RESULTS 

The Novel and the Familiar: 

How Do Students Approach CLAS Mathematics Tasks? 

Data on the ways in which students approach tasks on the CLAS, and 
more particularly, whether there are differences in how students approach 
novel open-ended and more familiar multiple-choice items, come principally 
from the observations of CLAS administrations, t*ie teacher interviews, and 
the student think-aloud interview protocols. 

• Observation data collected during the assessment itself were used to 
characterize students' approaches in terms of: how much time 
students spent working through open-ended problems, how much time 
they spent on the assessment in general, the kinds of questions 
students had during the assessment, and when — if at all — students 
utilized calculators on the CLAS. 

• Teacher interview data were used to clarify calculator use during the 
assessment by indicating how accessible were calculators in students' 
daily lives, what restrictions teachers put on calculator use in the 
classroom, and what kind of instruction on calculator use students 
had received. 

• Data from student interviews following the CLAS administration 
elucidated how students tackled open-ended, and multiple-choice CLAS 
tasks. Which aspects of solving the CLAS tasks did students perceive 
as important? Did students pursue mathematics-based reasoning to 
solve the tasks or did they use a trial-and-error approach? Did student 
guessing vary by the type of problem they were engaged in solving? 

This section strives not only to describe students' approaches to the CLAS 
in general, but also to compare responses across problem type and acrrss 
schools: Are there differences in students' approaches to open-ended and 
multiple-choice problems? Are these differences consistent with the claims 
made for alternative assessment? And are there differences in how students 
in different types of schools responded to the CLAS tasks? 

m 
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Students' Use of Time During the Assessment 

Ideally, open-ended problems are designed to engage students in in-depth, 
complex, and extended thinking, in contrast to typical selected response items. 
Because the CLAS was not intended as a timed test, and in fact teachers were 
instructed to "make special arrangements for students who are still 
productively engaged at the end of 45 minutes, providing additional time for 
them to complete their work," observation of the time students actually spent 
on the assessment gives some indication of their engagement level and the 
ease with which they completed the novel assessment. In addition, beca'ise an 
open-ended item appeared first on the assessment, observers were able to note 
how long students spent on that item— or at least whether students spent the 
time assessment developers had estimated was required for a thoughtful 
response. The extent to which students used the 15 minutes anticipated for 
such items provided another indicator of students' engagement in open-ended 
items. 

Table 4 shows the distributions of students in observed classes using at 
least 15 minutes to complete the first open-ended item. In general, most 
classrooms had 75-100% of their students spending at least 15 minutes on the 
first open-ended problem. However, results also indicate significant 
differences in schools serving different types of communities (x 2 (8) = 28.92, p = 
.0003). Whereas in 100% of the suburban classrooms observed, almost all of the 
students used at least the allotted 15 minutes to answer the first open-ended 
problem, such extended concentration by most students was observed in only 



Table 4 

Class Distributions: Percentage of Students Using at Least 15 Minutes to Answer First 
Open-Ended Problem by School Type (CLAS Administration Observation Results)* 





A few 


About 25% 


About 50% 


About 75% 


Almost 


Type of school 


students 


of students 


of student3 


of students 


all students 


Urban 


4.9 


12.2 


31.7 


14.6 


36.6 


Rural 


0 


4.3 


4.3 


17.4 


73.9 


Suburban 


0 


0 


0 


0 


100 


Totals 


2.4 


7.1 


16.7 


11.9 


61.9 



*p < .05. 
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37% of the urban classrooms. In nearly half the urban classrooms, 50% or 
more of the students were observed to have moved on earlier in the period. 
Students in rural classrooms more closely resembled the suburban students, 
with most students in about three-quarters of the classrooms using the full 15 
minutes to answer the first problem. 

Analysis of variance showed similar significant differences, F(2) = 15.47, 
p < .0001. On a scale of 1 ("No students in the class used the full 15 minutes") to 
6 ("Almost all students used the full 15 minutes"), the overall mean for all 
classrooms was 5.24; but, the mean for urban classrooms was 4.67, the mean 
for rural classrooms was 5.61, and the mean for suburban classrooms was 
6.00. An HSD Tukey test indicated significant differences between urban and 
rural schools (p < .05) and between urban and suburban schools (p < .05); no 
significant difference was found between the rural and suburban classrooms. 
Thus, students in urban classrooms appeared less engaged by and less 
involved in extended problem solving on the first open-ended question than did 
other students. 

Observers also were asked to estimate the percentage of students who 
completed the CLAS during the regular assessment period (see Table 5). 
Overall, in most classrooms, 75-100% of the students finished during the 
allotted time period. Significant differences across school types were again 
found, x 2 (12) = 31.87, p = .001. In almost all suburban classrooms observed 
(94%), most or all students finished the assessment during the regular 
assessment period, while in only two-thirds of the urban classrooms did 



Table 5 

Class Distributions: Percentage of Students Completing the CLAS within the Regular 
Assessment Period by School Type (CLAS Administration Observation Results)* 



Type of 




A few 


About 25% 


About 50% 


About 75% 


Almost 


school 


None 


students 


of students 


of students 


of students 


all students 


Urban 


5.3 


10.5 


2.6 


2.6 


13.2 


65.8 


Rural 


0 


4.3 


0 


17.4 


43.5 


34.8 


Suburban 


0 


0 


0 


0 


0 


94.1 


Totals 


2.6 


6.4 


1.3 


6.4 


19.2 


62.8 



*p < .05. 
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observers report that most students finished during this period. Rural schools 
showed the lowest completion rate, with most students finishing within the 
allotted time in only 35% of the rural classrooms observed. 

The results regarding time usage on the first open-ended problem may 
indicate urban students did not fully develop their x esponses to the open-ended 
task and perhaps were frustrated by it. Compared to students in other schools, 
urban students apparently had less to say in response to the first question and 
tended to rush through their response. Among other things, their haste may 
indicate a lack of background knowledge to fully respond to the items, 
insufficient experience with such items, and/or a lack of interest or motivation 
in the item. 

The differences that emerge between time spent on the first open-ended 
item and that spent to complete the entire assessment, at face value, seem 
contradictory. Although students in urban classrooms seemed to move more 
quickly through the first open-ended problem than students in suburban or 
rural classrooms, urban students were less likely to complete the full 
assessment in the 45 minutes generally allocated to it. Where did urban 
students spend their time? Is it possible that these students spent much more 
time on the multiple-choice problems, indicating they had greater difficulty 
with these items than students in other schools? As shown later in this paper, 
the trial-and-error approach some students used to solve multiple-choice items 
may have caused them to spend a long time on such problems. It is also 
possible that students revisited their responses to the open-ended tasks later on 
in the assessment period, and worked back and forth between the open-ended 
and multiple-choice items. While it also is conceivable that teachers at the 
different schools types reacted differently to the time constraint, influencing 
how comfortable students felt in continuing to work past the allotted time 
period, it does appear that students in rural and suburban schools had more 
efficient strategies for completing the assessment. 

Questions Arising During the Assessment Administration 

The kinds of questions students have during the administration of a new 
assessment provide an estimate of what difficulties they may be experiencing. 
While data collected during classroom observations indicated that students 
overall did not ask many questions during the CLAS (mean number of 
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questions per classroom = 1.5), there were significant differences across 
schools (F(2) « 3.21, p « .045) (see Table 6). Suburban students asked the most 
questions (mean of 2.2 questions during the assessment), followed in frequency 
by urban students (mean of 1.4 questions) and rural students (0.8 questions on 
average). 

For most types of questions (requests for more information, questions 
about specific math content, negative comments, and so on), no differences 
between schooL: were found. Differences, however, did arise in two areas. 
First, students in suburban classrooms were far more likely than other groups 
to pose procedural questions, such as "Where do I work the problem out?" or 
"Where do I start the multiple-choice?" (x 2 (10) a 25.01, p « .005). While posed in 
less than one-third of the urban classrooms and in about 40% of the rural 
classrooms, procedural questions arose in almost two-thirds of the suburban 
classrooms (see Table 6). 

Contrary results were found regarding questions about a key term in the 
"thinking curriculum" for mathematics. The second open-ended question on 
the common form asked students, among other things, to state their 
assumptions. As Table 6 indicates, students in less than 5% of the suburban 
classrooms raised questions about the meaning of this term, but in about 40% 
of urban and rural classrooms, students asked for clarification (x 2 (8) » 15.43, 
p = .05). Clearly, since questions in general tended to be raised less often in 
rural and urban classrooms than in suburban ones, the different frequency on 
the "assumption" issue cannot be attributed to students 1 propensities for 
asking questions. Rather, these findings seem to indicate that relative to 



Table 6 

Questions During the CLAS Administration by School Type (CLAS Administration 
Observation Results)* 



Type of 
school 


Mean number of 
total questions 


% Classes with 
procedural questions 


% Classes with 
"assumption" questions 


Urban 


1.44 


29.3 


43.9 


Rural 


.83 


39.1 


39.1 


Suburban 


217 


63.6 


4.5 


*p < ,05, 
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suburban students, students in rural and urban schools are less familiar with 
an important concept in mathematical thinking and problem solving, making 
and using assumptions. Although this may be a problem of technical 
vocabulary, as opposed to underlying concept understanding, it is clear that 
some urban and rural students were at a disadvantage when solving the 
second open-ended task. 

Students' Access to and Use of Calculators During Administration 

New ideas in the teaching of mathematics emphasize the importance of 
using tools to help solve problems. In addition, concern for equity demands 
that mathematical tools such as calculators be available and equally accessible 
to all students, not just students whose parents can afford them, and that all 
students be able to approach mathematical problems with such tools in hand. 
Teacher interviews supplied data on the availability of calculators, students' 
previous experience with calculators, and the constraints teachers place on 
calculator use in the classroom. In addition, classroom observations provided 
information on the extent to which calculators actually were used during the 
assessment. 

According to teacher reports, there were no significant differences across 
schools in the number of students who have their own calculators: Teachers 
across all schools claimed that most students have calculators. However, there 
did appear to be significant differences across school types in the sources of 
these tools. Urban teachers reported that their schools provided calculators 
for almost all students (mean of 4.6 on a 5-point scale where 6 indicates that 
calculators are provided for all students, and 0 indicates no calculators are 
provided), while rural and suburban schools tended to provide calculators for 
only some students (mean of 3.0) (F(2) = 5.94, p = .008). In contrast, teachers in 
suburban schools acknowledge that more of their students brought calculators 
to school from home (mean of 4.3) than those in rural (2.3) and urban (3.0) 
schools (F(2) = 3.24, p = .056) (see Table 7). 

When asked about their students' competence in using calculators, 
teachers claimed that a majority of their students were competent, with no 
significant differences found among schools. In regards to classroom policies 
on the use of calculators, more than half the teachers (54.3%) indicated they 
allowed their students unrestricted use of calculators. Of those teachers who 
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Table 7 

Means Value, for Calculator Availability and Calculator Use During the 
Assessment by School Type (Teacher Interview and CLAS Administration 
Observation Results) 



Type of school 


Calculators 
provided 
by school 3 


Calculators 
brought from 
home a 


Students using 
calculators on 
theCLAS b 


Urban 


4.6* 


3.0 


2.7* 


Rural 


3.0* 


2.3 


2.8* 


Suburban 


3.0* 


4.3 


4.8* 



Note. 1 = No students, 5 = Almost all students. 
a Based on teacher interview data. 
b Based on observation of CLAS administration. 
*p < .05. 



restricted calculator use, almost all (94.1%) allow students to use calculators 
on tests. Again, no significant differences were found across schools. 

Observation results on availability of calculators similarly indicated no 
differences across different school types: Almost all students, according to 
observers, had calculators available to them for the CLAS assessment. During 
the assessment itself, however, observation data suggested that students in 
suburban classrooms were significantly more likely than those in rural or 
urban schools to approach CLAS tasks vith calculators in hand (F(2) = 22.51, 
p < .001) (see Table 7). In suburban classrooms, almost all students used 
calculators in completing the CLAS, while in urban and rural classrooms only 
some students did so. 

Observers also noted in which parts of the assessment students were most 
likely to use calculators — the open-ended or the multiple-choice portions. 
Consistent with the previous findings, students in suburban classrooms were 
more likely than those in other classrooms to use calculators across all 
problems (x 2 (6) = 23.76, p = .0006) (see Table 8). In almost all (91%) of the 
suburban classrooms, students were observed using calculators equally for 
both the open-ended and the multiple-choice sections of the assessment, while 
such uniform use occurred in only about half of the urban clasorooms and 
about a quarter of the rural classrooms. 
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Table 8 

Percentage of Classrooms Showing Different Patterns of Calculators Use on CLAS Tasks 
by School Type (CLAS Administration Observation Results)* 







Used mainly on 


Used mainly on 


Used on 


Type of 


Calculators 


open-ended 


multiple-choice 


both items 


school 


not used 


items 


items 


(half-and-half) 


Urban 


5 


22 


22 


51 


Rural 


0 


39 


35 


26 


Suburban 


0 


0 


9 


91 



*p < .05. 



Student Perceptions of Performance Criteria 

Because the CLAS contains both novel open-ended and more familiar 
multiple-choice problems, it provided an opportunity to examine students' 
understandings of the performance criteria for each question type. Clearly, 
how students approach a given task will be influenced by their expectations for 
what is required. In the retrospective interviews, students were asked what 
they thought their teachers would he looking for as they scored student 
responses to open-ended problems — how would their responses be graded? 
Parallel questions were posed about the multiple-choice problems. Responses 
were coded to indicate whether or not students mentioned any of the following 
dimensions as important in grading: (a) the correct answer, (b) the steps 
students used to solve the problem, (c) students' use of graphs, charts, or 
diagrams, and (d) the depth of students' explanation and understanding. 
Descriptive statistics were calculated to characterize student perceptions of the 
criteria, and these perceptions were compared across type of task. 

Table 9 shows the distribution of students who mentioned each of the four 
criteria. For novel open-ended items, half the students mentioned the 
importance of the quality or depth of their explanations; 46% mentioned 
attention to their use of diagrams, graphs and other visuals; 29% mentioned 
that the steps of their solutions would be important; and 26% thought the 
correct answer was an important element in scoring. In contrast, when asked 
about the multiple-choice items, 43% of the students indicated that scorers 
would be looking for the correct answer, while 35% mentioned the steps used to 
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Table 9 

Percentage of Students Mentioning Each of the Four Grading Dimensions 
(Student Interview Results) 

Type of task Correct Steps Diagrams Explanation 

Open-ended 2S 29 46 50 

Multiple-choice 43 35 0 17 



solve the problem (perhaps recognizing that if their method was not correct, 
they were unlikely to get the right answer). Diagrams were not mentioned by 
any of the students, and the importance of explanation was mentioned by only 
17% of the students. 

In addition to the information coded from the student interviews, students 
also were asked directly whether they thought or did anything differently when 
responding to open-ended problems compared to multiple-choice problems (see 
Table 10). Almost two-thirds of the students (64.6%) reported they approached 
open-ended tasks differently. When asked why they reported differences, 
almost half the students replied that they had to explain their thinking and 
over 40% reported that they had to think harder on open-ended items. Other 
answers (each mentioned by about 10% of the students) included the need to 
create their own answers, the use of diagrams, and the lack of given responses 
from which to choose. There were no apparent differences by school type. 

It appears then that students do perceive different expectations for their 
responses on open-ended versus multiple-choice problems. Multiple-choice 
items are associated with the use of appropriate algorithms and the 
determination of a correct answer; open-ended items— on the other hand — are 

Table 10 

Percentage of Students Reporting Reasons for Approaching Open- 
Ended Items Differently Than Multiple-choice Items (Student 
Interview Results) 



Own 


No choices 


Use 


Qive 


Think 


creation 


given 


diagrams 


explanations 


harder 


11.3 


12.8 


10.5 


42.8 


36.3 
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allied with the use of diagrams and the need to explain one's results. Students 
apparently are aware that open-ended tasks require a different type of 
approach than do the more familiar multiple-choice tasks. 

Lines of Reasoning 

Another aspect of students' approaches to CLAS tasks is the line of 
reasoning students use when solving the tasks. Do students pursue a 
mathematics-based reasoning approach to solve a given problem or do they use 
a nonmathematically-oriented trial-and-error or guessing approach? Are 
there differences in how students approach novel open-ended tasks and how 
they approach familiar multiple-choice tasks? 

Student retrospective interview responses were coded for type of reasoning 
students used, and descriptive statistics were calculated. "Mathematics-based 
reasoning" was defined as that which utilized disciplinary concepts (rightly or 
wrongly) or strategic lines of reasoning based on mathematical thinking — for 
example, students who reasoned about a problem "Well, the area of the area of 
the larger square minus the smaller square should give you the shaded area 
and to get area from perimeter you . . . n . Random trial-and-error approaches, 
in contrast, were nonlogical from a mathematics perspective — for example 
taking the numbers in a problem and trying to play with them in some way to 
come up with an answer given in the multiple choice alternatives: 

Well, first I tried to multiply them, but that wasn't an answer, but 
that wasn't an answer, then I thought about adding them but that 
din't work either, so then I subtracted them which gave me 6 and 
then I divided by i because their were two of them and the number 
3 was an answer. 

These latter types of responses were combined with students who admitted 
guessing. For the open-ended problems, students overwhelmingly followed 
some mathematics-based reasoning approach (whether correct or incorrect) 
rather than using a trial-and-error or guessing approach: Only 1% of the 
students' responses were coded as guesses. In contrast, 34% of the students 
used a trial-and-error or guessing approach on the multiple-choice items. 
These results are displayed in Table 11. There were no apparent differences by 
school type. 
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Table 11 

Percentage of Students Using Guessing or Mathematics-Based Reasoning 
Approach (Student Interview Results) 





Used trial-and-error or 


Used mathematics-based 


Type of task 


guessing approach 


reasoning approach 


Open-ended 


1 


99 


Multiple-choice 


34 


66 



Students also were asked in the interviews whether they guessed on tests. 
Over 80% of the students reported guessing on tests, and the results, shown in 
Table 12, revealed significant differences across school types, x 2 (2) = 7.88, p = 
.02. Two-thirds of both rural and suburban students reported guessing on 
tests, compared to over 85% of urban students. When asked on which type of 
problem — open-ended or multiple-choice — they guessed more, students 
reported guessing more on multiple-choice problems, by a margin of 82.6% to 
14.2%, with 3.2% of the students reporting there was no difference (see 
Table 13). No significant differences were found across school type for this 
response. When asked why they guessed more on the indicated problem, over 
60% of the students reported that they guessed more on multiple-choice 
problems because the answer choices for these problems were available. 
Coding of students' comments during the retrospective interviews indicated 
that 32% of all students mentioned using the multiple-choice response 
alternatives as prompts to help them solve the problem. 

Table 12 

Percentage of Students Who 
Reported Guessing on Tests by 
School Type (Student Interview 
Results)* 



Type of school 


Guess on tests 


Urban 


85.7 


Rural 


66.7 


Suburban 


68.6 


Total 


80.8 


*p < .05. 
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Table 13 



Percentage of Students Who Reported 
Guessing More by Type of Problem 
(Student Interview Results) 



Type of task 



Guess more 



Open-ended 
Multiple-choice 
No difference 



14.2 



82.6 



3.2 



Students reported a variety of ways they used the given alternatives to 
prompt their responses: Some students reported using the alternatives to 
check their work; others indicated selecting the "closest" alternative to the 
answer they had computed; and still others reported guessing from the given 
alternatives. A number of students also indicated they used the possible 
alternatives as a starting point, working backwards from these alternatives to 
the initial problem. 

Summary 

From the data presented, we can begin to paint a picture of how students 
approach the mathematics tasks found on the California Learning 
Assessment System. Our observations show differences in how long students 
spend on the open-ended tasks on the one hand, and on the assessment as a 
whole on the other. Students in low-SES urban schools appear to move 
through the open-ended items more quickly, but spend a longer time overall 
than students in other schools in the sample. In addition, students in low-SES 
urban schools experienced greater difficulty with the definition of a key term in 
mathematical thinking used in the CLAS directions ("assumption"), but asked 
fewer questions overall than other students. That they report guessing more 
on tests is perhaps another indicator that urban students are less prepared for 
the assessment than their peers. Fortunately, calculator accessibility and 
preparation was high for all students; however students in higher SES 
suburban schools tended to use their calculators throughout the assessment 
whereas students in urban schools tended to use them more sporadically. 
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Turning to comparisons across task types, students seem to understand 
the differences in approach necessitated by open-ended versus multiple-choice 
problems. They know that open-ended problems emphasize students' use of 
explanatory materials (e.g., graphs, charts, diagrams, and the quality of their 
explanations) and focus less on algorithms and the correct answer. Students 
seem to follow a mathematics-based reasoning approach — correct or 
otherwise — in responding to open-ended items; they approach multiple-choice 
items both logically and by using trial-and-error or guessing. Students use 
these alternatives in various ways to prompt their responses, marking 
different approaches to solving multiple-choice tasks than the more novel 
open-ended tasks. 

What Are Students' Attitudes 
Toward Different Assessment Tasks and Mathematics? 

As indicated in the introduction, the literature surrounding alternative 
assessment suggests that students will find alternative assessments more 
meaningful and motivating than traditional multiple-choice tasks, and 
furthermore, that students who engage in authentic tasks are likely to be more 
motivated to learn in school. Because the Math CLAS uses both open-ended 
and multiple-choice questions, it provided an opportunity to investigate student 
attitudes towards these two different types of questions and toward math in 
general. Student surveys asked students a series of questions about their 
comparative reactions to open-ended and multiple-choice tasks and their 
attitudes toward mathematics. In addition, the retrospective interviews 
queried students about their relative perceptions of the two types of tasks, the 
reasons for their responses, their feelings about what they liked best and least 
about open-ended problems, and what advice they would offer another student 
who had to prepare for an assessment like the CLAS. Student responses were 
c : npared both across type of problem and by type of school. 

Student Attitudes Towards Open-Ended Versus Multiple-Choice Problems 

Students find open-ended tasks more challenging. Over half the students 
surveyed (55.2%) felt that open-ended problems made them try harder than 
multiple-choice questions, while only 11.1% reported they had to try harder on 
multiple-choice questions. (The remainder attributed no difference in effort to 
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the two types of questions.) In addition, chi-square analysis indicated 
statistically significant differences by type of school, with students in rural 
schools less likely than other students to believe that open-ended tasks required 
more effort, x 2 (4) = 13.75, p = .008 (see Table 14). 

Students who participated in the retrospective interviews were even more 
united in their belief that open-ended questions are harder or more 
challenging to answer than multiple-choice questions. The vast majority 
(83.4%) reported that open-ended questions are more challenging, including 
94.3% of the suburban students, 83.3% of the rural students, and 70.2% of the 
urban students. Asked why, nearly half of the students (44.7%) mentioned that 
open-ended questions are more challenging because they cause students to 
think harder, are more difficult, or are more complicated to answer. In 
addition, approximately one-third (35.6%) of the students pointed out that open- 
ended questions required them to explain their answer by showing their work, 
or to communicate their math knowledge verbally. About one-fifth of the 
students stated that open-ended questions are more challenging because 
students had to create an answer on their own (21.7%) or that alternative 
answers were not provided (21.9%) (see Table 15). 

Students' perceptions of the relative challenge in open-ended versus 
multiple-choice tests, in short, mirror the intentions of proponents and 
developers of alternative assessments: Such items apparently require students 
to actively accomplish complex tasks. In addition, students may feel more 
challenged because alternative assessment is a relatively new approach, and 
thus students are less familiar with these types of questions and the processes 
they entail. 



Table 14 



Percentage of Students Reporting the Type of Question That Causes 
Them to Try Harder by School Type (Student Survey Results)* 



Type of school 


Multiple-choice 


Same 


Open-ended 


Urban 


13.1 


30.6 


56.3 


Rural 


13.0 


38.9 


48.1 


Suburban 


4.5 


33.4 


59.1 


Total 


11.1 


33.7 


55.2 



*p < .05, 
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Table 15 

Percentage of Students Giving Reasons Why Open-Ended Items 
Make Students Think Harder or Are More Challenging Than 
Multiple-Choice Items (Student Interview Results) 



Harder 


Explanation 


Own creation 


No choices 


44.7 


35.6 


21.7 


21.9 



Students find open-ended tasks more interesting. As Table 16 shows, 
slightly more than half of the students surveyed— and almost two-thirds of 
those expressing a preference — indicated that open-ended questions were 
more interesting to solve than multiple-choice questions. When analyzed by 
type of school, a chi-square test shows that rural students are less likely than 
other students to express preferences for open-ended questions (x 2 (4) = 18.97, p 
= .0008). Asked what they liked best about open-ended questions, interviewed 
students reported that they liked being able to create their own answer and 
having an opportunity to explain their answer; they also liked that the 
questions were challenging (see Table 17). 



Table 16 

Percentage of Students Reporting the Type of Question They Find Most 
Interesting by School Type (Student Survey Results)* 



Type of school 


Multiple-choice 


Same 


Open-Ended 


Urban 


27.4 


20.6 


56.0 


Rural 


41.0 


19.3 


39.8 


Suburban 


23.4 


15.5 


60.9 


Total 


29.4 


19.2 


51.4 



*p < .05. 



Table 17 

Percentage of Students Reporting What They Like Best 
About Open-Ended Problems (Student Interview Results) 



Own creation 


Explain 


Harder 


14.0 


26.9 


28.6 
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Claims which state that open-ended problems are more relevant and thus 
more motivating to students than traditional problems are bolstered by these 
results. In addition, these findings are encouraging since alternative 
assessments strive to tap higher order cognitive processes or problem-solving 
skills and to encourage students to create or produce a response. 

Liking is another matter. While the majority of students surveyed (61%) 
reported they were not frustrated by problems with more than one answer, 
only about one-third of the students surveyed (38.6%) stated that they liked 
problems with no obvious solution. In addition, only 17.4% agreed that they 
liked problems that take a lot of time to solve. No significant differences were 
found by type of school. 

Confirming these responses, survey results indicate that more students 
prefer multiple-choice questions (60.7%) compared to the 17.4% who reported 
liking open-ended problems better (see Table 18). While still a minority, 
students in suburban schools were more likely to express a preference for 
open-ended tasks than other students (x 2 (4) = 10.39, p = .034). 

Similar patterns were found in the interview results: Just over half of the 
students interviewed (54.4%) stated they liked multiple-choice tasks better than 
open-ended tasks. The reasons these students gave for liking multiple-choice 
questions included that these questions were easier (57.6%) and that the 
choices were given (40.7%). Reasons given for disliking open-ended questions 
included: the need to explain answers, problems in understanding the 
questions, the difficulty of the questions, and the lack of clarity or enough 
information given in the questions (see Table 19). 



Table 18 

Percentage of Students Reporting the Type of Question They Like 
Better by School Type (Student Survey Results)* 



Type of school 


Multiple-choice 


Same 


Open-ended 


Urban 


62.3 


22.7 


15.0 


Rural 


60.5 


24.1 


15.4 


Suburban 


56.9 


17.8 


25.3 


Total 


60.7 


21.9 


17.4 



*p < .05. 
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Table 19 

Percentage of Students Reporting What They Like Least About Open-Ended 
Questions (Student Interview Results) 









Not clear/Not enough 


Harder 


Explain 


No understanding 


information given 


42.2 


18.2 


9.9 


7.8 



Thus, it seems that although students report open-ended items to be more 
interesting, they still prefer multiple-choice items. Students' beliefs that 
multiple-choice problems are easier stem partly from the availability of 
response choices for these types of problems; in addition, multiple-choice 
problems may be perceived as easier because of students' familiarity with 
these types of items as compared to the relative newness of open-ended items. 

Students find mxiltiple-choice items easier to understand. Since students 
frequently felt that open-ended questions were more difficult, it is not 
surprising to find that almost two-thirds of the students interviewed (64.9%) 
felt that in multiple-choice questions it was easier to understand what to do 
than in open-ended questions. Of the students reporting that multiple-choice 
questions make it easier to understand what to do, they attributed this ease to 
the choices offered (38.7%), the questions being easier (25.3%) or stated more 
clearly (17.9%), and the lack of a requirement to explain their answers (11.9%) 
(see Table 20). Students' responses, in fact, are similar to some of the 
criticisms that have been mounted against multiple-choice questions (e.g., 
neglect of complex thinking and problem solving). The familiarity of the 
multiple-choice format may also play a role in students' beliefs that these types 
of questions make it easier to understand what to do. 

Table 20 

Percentage of Students Reporting Why They Feel Multiple- 
Choice Questions Make It Easier to Understand What to Do 
Than Open-Ended Questions (Student Interview Results) 



Choice offered 


Clear 


Easier 


No explain 


38.7 


17.9 


25.3 


11.9 
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Consistent with their opinions about which type of problem makes it 
easier to understand what to do, more surveyed students felt they did better on 
multiple-choice questions (68.8%), compared to those reporting they did better 
on open-ended questions (14.0%, x 2 <4) = 10.00, p = .04; see Table 21). The 
familiarity of the multiple-choice format may again account for part of the 
reason for this judgment. 

Students hold mixed opinions on which items best show their 
knowledge/abilities. The student survey and the student interview results 
yielded somewhat contradictory views of students' feelings about whether 
multiple-choice or open-ended questions best let them show what they know 
about math. Survey results found students about evenly split between multiple- 
choice and open-ended tasks, with significant differences found among the 
school types. Chi-square results, X 2 (4) = 21.15, p = .0003, suggest that rural 
students are more likely to favor multiple-choice items: Over half the rural 
students surveyed indicated that multiple-choice items let them show their 
mathematics knowledge better than open-ended items. Urban students are 
more evenly split, while suburban students report open-ended items best let 
them show what they know about math (see Table 22). 

However, interviewed students (who reviewed their responses to the 
multiple-choice and open-ended tasks prior to responding to this question) 
were more likely to believe that open-ended questions better enabled them to 
show what they know. Over half of the students (55.7%) so indicated, with no 
significant differences found by type of school. Over two- thirds of these 
students stated that open-ended questions allowed them to explain their 



Table 21 

Percentage of Students Indicating the T;n>« of Question on Which Tlwy 
Felt They Did Better (Student Survey Re- suits)* 



Type of school 


Multiple-choice 


Same 


Open-ended 


Urban 


70.0 


15.3 


14.8 


Rural 


69.6 


14.3 


16.1 


Suburban 


65.3 


24.4 


10.2 


Total 


68.8 


17.2 


14.0 



*p < .05. 
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Table 22 

Percentage of Students Reporting What Type of Question Best Lets Them 
Show What They Know by School Type iStudent Survey Results)* 



Type of school 


Multiple-choice 


Same 


Open-ended 


Urban 


38.4 


24.1 


37.5 


Rural 


54.9 


20.4 


24.7 


Suburban 


31.8 


26.1 


42.0 


Total 


40.4 


23.8 


35.8 



*p < .05. 



answers. Additionally, students reported that open-ended questions best let 
them show what they know about math because the questions were more 
challenging (20.4%), they did not provide choices for answers (12.4%), and 
students were allowed to create their own answers (8%). 

Inconsistent findings between survey and interview responses are 
puzzling. It may simply be that students are unsure which type of question 
best lets them show what they know because they are as of yet unfamiliar with 
how understanding of open-ended problems is assessed. In contrast, the 
manner in which multiple-choice responses are scored is clear and straight- 
forward. 

Students give a variety of advice. At the close of the interview, students 
were asked what advice they would offer another student who had to prepare 
for an assessment like the CLAS. Students volunteered a variety of strategies, 
typically drawing from the preparation they themselves had experienced (see 
Table 23). Students most frequently mentioned the importance of studying 
course material, including practicing homework problems and reviewing 
notes. Students also mentioned metacognitive and test-taking strategies such 
as re-reading the assessment problems and checking their answers. In 
addition, students stated they would advise peers to pay attention in class, to 
listen to the teacher, to work hard in class, to make an effort, to stay calm, and 
to concentrate. No significant differences in advice given were found by school 
type. 
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Table 23 

Percentage of Students Mentioning Advice 
to Peers (Student Interview Results) 



Type of advice 


Mention advice 


Study 


60.7 


Metacognitive 


16.5 


Class 


14.5 


Effort 


12.9 


Relax 


6.2 


Think 


9.3 


Take time 


3.6 


Easy 


2.1 


Hard 


1.6 



Student Attitudes Towards Mathematics 

Based on the survey results, students generally expressed modestly 
positive attitudes toward mathematics. The majority of the students agreed 
moderately or very much with the statements that they liked math (60.9%), and 
that they were good in math (65.6%), and they were almost unanimous in their 
belief that math would be useful to them in the future (91.2%). There were no 
apparent differences in student attitudes associated with different types of 
schools. 

Although students appeared to have positive attitudes toward 
mathematics and their mathematical abilities, they expressed less 
enthusiasm for their mathematics instruction. In response to the statement 
that learning math is mostly memorizing, the majority of students in urban 
and rural schools agreed moderately or very much; significantly fewer 
students from suburban schools held such a view (45%, % 2 (6) - 16.68, p - .01). 
In addition, the majority of the students agreed that they did not like 
computational problems (52.9%) or projects that require math (63.2%) (see 
Table 24). With the exception of the item on memorization, no significant 
differences for these attitudes were found across school types. 
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Table 24 

Students' Attitudes Toward Mathematics: Percentage of Students Expressing 
Moderate or Strong Agreement by School Type (Student Survey Results) 



Student attitude 


Urban 


Rural 


Suburban 


Like math 


62.3 


61.7 


56.8 


Good in math 


64.4 


64.2 


69.9 


Useful in future 


91.2 


90.5 


92.0 


Mostly memorizing 


56.1* 


59.2* 


44.8* 


Do not like computa- 
tional problems 


56.1 


45.1 


52.0 


Do not like projects that 


63.6 


58.7 


66.5 



require math 
*p < .05. 

Overall Attitudes Toward Mathematics and Open-Ended Items 

As mentioned earlier, individual attitude items from the student survey 
were aggregated into two subscales. The first subscale was composed of items 
related to students' attitudes toward mathematics. The items that made up 
this scale were taken from students* responses to a question that asked them 
how well they agreed with several statements about mathematics. The items 
used for the first subscale were: 

• "I like to do math problems that don't have an obvious solution." 

• "I think math will be useful to me in the future." 

• "I like to do computation problems." 

• "Learning math is mostly memorizing." 

The second subscale was a summary indicator of students* attitudes 
toward open-ended items. These items asked students to choose between open- 
ended and multiple-choice items where 1 was "Multiple-choice," 2 was "About 
the same," and 3 was "Open-ended" on the following items: 

• "Which type of question did you find most interesting?" 

• "On which type of question do you think you did better?" 

• "Which type of question did you like better?" 
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Confirming patterns in results of individual items, there were no 
differences by type of school in students' attitudes toward mathematics, but 
there were differences by school type in students' liking of open-ended 
problems. As Table 25 shows, students in rural schools were significantly less 
positive in their attitudes toward open-ended items, while students in 
suburban schools were the most positive in their reactions to such items, F(2) = 
8.46,/?<.001. 

There war very little relationship between students' attitudes toward 
mathematics and their liking of open-ended items (r = .14). The relationship 
between students' attitudes and their performance will be investigated further, 
when performance data from the state become available. 

Summary 

In general, students' perceptions of open-ended items are consistent with 
major aims of proponents and designers of alternative assessment: Students 
find such items more interesting and challenging than multiple-choice items 
and recognize that open-ended items require them to think harder, explain 
their thinking, and communicate their understanding of mathematical 
knowledge. At the same time, however, students do not necessarily like such 
challenges. In fact, students express a preference for multiple-choice items: 
They find multiple-choice items easier to understand and believe that they 
perform better on such items. Students in the rural schools in our sample 
were the least positive about open-ended items, while suburban students were 
relatively the most positive. These preferences may in part be due to the 
relative newness of open-ended items as compared to the comforting 
familiarity of multiple-choice items. Turning to student attitudes toward 



Table 25 

Students' Overall A ttitudes Toward Mathematics and Open-ended 
Items: Summary Subscales by School Type (Student Survey Results) 



Overall attitude 


Urban 


Rural 


Suburban 


Like math 


11.05 


11.35 


11.08 


Like open-ended items 


7.18* 


6.66* 


7.65* 


*p < .05. 
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mathematics, students in all schools were only moderately positive about 
mathematics, but quite positive about the value of the discipline in their future. 
No differences were found across school type in these general attitudes. 

What Is the Nature of Students' Opportunity to Learn 

the CLAS Content and Skills? 

One of the study's key issues focused on students' opportunity to learn the 
complex mathematical thinking, communication, and problem-solving skills 
that the CLAS seeks to assess, and more specifically whether students in all 
schools— regardless of background and SES level— have equal opportunity to 
learn that which is assessed. Included in our definition of "opportunity to 
learn" were access to resources, such as qualified teachers and appropriate 
instructional tools; access to the types of instructional content and processes 
likely to help students develop required knowledge and skills; and direct 
preparation and practice for CLAS-type assessments. We expected students 
who had different classroom experiences to have different perceptions of their 
own preparedness for the CLAS and to show different achievement levels on 
the assessment itself. We thus compared students' opportunity to learn across 
school types to address this hypothesis. 

Access to Quality Resources 

With regard to access to quality resources, the study gathered data from 
teachers on their preparation to teach mathematics, including their 
undergraduate fields, whether or not they were credentialed to teach 
mathematics, years of experience teaching mathematics, participation in 
recent professional development that would likely prepare them in the content 
and instructional practices of a "thinking curriculum" in mathematics, and 
preparation for the CLAS itself. Also examined was access to appropriate 
instructional materials, including calculators and recent textbooks. 

Teacher preparation. Although less than half of our teacher sample had 
either majored (22.2%) or minored (26.0%) in a mathematics field (including 
engineering and computer science), the majority of teachers (66.7%) were 
credentialed to teach mathematics. Seventy-eight percent of the urban 
teachers, 60% of the suburban teachers, yet only 25% of the rural teachers 
indicated that they had such certification. Similarly, suburban and urban 
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teachers were more likely than rural teachers to have majored or minored in 
mathematics as undergraduates. No rural teachers held undergraduate 
degrees in mathematics, and only one was certified to teach mathematics. 

Similar patterns emerged when data on in-service education were 
examined. Urban and suburban teachers were more likely to have 
participated recently in extended professional development: Sixty-seven 
percent of the urban teachers and 80% of the suburban teachers reported 
spending more than 35 hours over the last three years in in-service education 
in mathematics or the teaching of mathematics, while only 25% of the rural 
teachers reported that level of activity. No differences were found across school 
type in years of teacher experience teaching mathematics, with a mean of 11 
years for the total sample. 

Table 26 shows teachers' responses when questioned about their specific 
preparation for the CLAS. Teachers in general were not highly confident 
about their preparation to teach CLAS-type objectives, with no more than half 
the teachers representing each school type expressing that they were "very 
weir prepared. 

Instructional resources. Because the National Council of Teachers of 
Mathematics (NCTM) standards and the California Curriculum Framework 
in Mathematics are relatively new, it is unlikely that older texts are well 
aligned with the reform ideas of the new standards. Recency of texts thus can 
be seen as an important indicator of access to relevant instruction 
opportunities. In this regard, teachers' reports of their primary textbooks 
indicated students .a urban classrooms were less likely to have recent texts 
than those in other schools in our sample (F(2) = 4.11, p = .03). 



Table 26 

Percentage of Teachers Expressing Preparation to Teach the 
CLAS by School Type (Teacher Interview Results) 



Type of school 


Not well 


OK 


Very well 


Urban 


8.7 


60.9 


30.4 


Rural 


16.7 


33.3 


50.0 


Suburban 


16.7 


33.3 


50.0 
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Access to calculators* Access to calculators is another indicator of the 
availability of instructional tools that reflect NCTM standards and the 
California framework. As described in the section on students' approaches to 
the CLAS, study data suggest no major differences between schools in 
students' access to basic calculators, although urban schools were more likely 
to provide them for students than were rural and suburban schools, and 
students in suburban schools were more likely to bring them from home. Of 
note is that for all types of schools, over 90% of the students have calculators at 
home. However, there is a difference in the type of calculators students have 
available to them at home: 62.7% of the suburban students have scientific 
calculators (as opposed to simple calculators) at home while only 44.5% of the 
urban students and 31 ,5% of the rural students have scientific calculators at 
home (% 2 (2) = 24.47, p < .0001). Although scientific calculators are not required 
for the CLAS, the availability of sophisticated calculators may indicate more 
familiarity and ease of use with such tools. 

Access to Learning Opportunities Appropriate to CLAS-Type Objectives 

Perceived fit between instructional practices and the CLAS. Chi-square 
analysis of the teacher interview results suggest that teachers in suburban 
schools were more confident about the fit between their teaching practices and 
the content of the CLAS than are teachers in other schools (x 2 (10) = 25.37, p =s 
.005). Asked how well their instruction aligned with the material on the CLAS 
assessment, two-thirds of the suburban teachers said that they felt their 
classroom instruction (including texts, teaching, and assignments) was an 
"OK" or "Excellent" match with the CLAS assessment. However, less than 
half of the urban teachers (47.8%) and none of the rural teachers felt their 
practices matched this strongly. 

Nonetheless, rural and urban teachers were more likely than suburban 
teachers to report that their students keep math portfolios — one of the 
hallmarks of the innovative practice because they are thought to encourage 
diversity of mathematics work, including math projects, writing, and 
investigations. Eighty-seven percent of urban teachers and 83.3% of rural 
teachers so reported, while only two (33.3%) of the suburban teachers reported 
having their students keep math portfolios (x 2 (2) = 7.92, p = .02). 
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Student preparation for concepts assessed on the CLAS. Students and 
teachers were asked to gauge the extent to which their classes had prepared 
for some of the math concepts included on the CLAS— focusing particularly on 
the content areas included on the common CLAS form that was used for the 
retrospective student interviews. Students were asked how well prepared 
they thought they were for: fractions, area, perimeter, graphing data, 
distance/time problems, and ratios. Similarly, teachers were asked how much 
class time was spent on these same areas. As Tables 27 and 28 show, students 
and teachers alike seem to agree that students were at least somewhat 
prepared in each of these areas. Note that both teachers and students report 



Table 27 

Percentage of Classes That Spent More Than Six Class Sessions on 
Content Areas by School Type (Teacher Interview Results) 



Content area 


Urban 


Rural 


Suburban 


Fractions 


75.9 


100.0 


100.0 


Area 


47.8 


83.3 


50.0 


Perimeter 


40.9 


66.7 


33.4 


Graphing data 


77.8 


66.7 


100.0 


Distance/time 


34.7 


83.4 


50.0 


Proportional reasoning 


73.9 


66.7 


83.3 



Table 28 

Mean Comparisons of Student Ratings of Their Preparation in 
Various Content Areas by School Type (Student Survey Results) 



Content area 


Urban 


Rural 


Suburban 


Fractions 


2.70 


2.57 


2.86 


Area 


2.34 


2.24 


2.62 


Perimeter 


2.25 


2.34 


2.65 


Graphing data 


2.32 


2.47 


2.54 


Distance/time 


2.16 


2,27 


2.36 


Ratios 


2.10 


1.94 


2.59 



Note. 1 = Little or none, 3 = Very well. 
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relatively high attention to fractions and graphing problems and relatively less 
to distance/time problems (except rural teachers). While the patterns are 
somewhat irregular for teacher reports, for the most part students in the 
suburban schools tend to feel that they are better prepared in these content 
areas. 

Teaching and instructional strategies which build complex thinking. 
Alternative assessment is intended to emphasize open-ended problems that 
require not only a solution but also an explanation of how the student arrived 
at such a solution and thus that value both complex mathematical thinking 
and communication. Both students and teachers were asked how often they 
engage in instructional practices that are associated with the development of 
these skills. Regarding these, Table 29 shows that a majority of students 
reported often solving word problems, solving problems that require thinking 
and that can be solved in more than one way, and using calculators. Students 
were less likely to report working on problems for which they must explain 
their thinking; that take at least a week to complete; that reflect real-life 
problems; for which they use rulers, blocks or solids; or that require oral 
presentations. For comparison purposes, students also were asked how often 



Table 29 

Mean Comparisons of Student Frequency Ratings of Their Engagement in Specific 
Activities by School Type (Student Survey Results) 



Activity 


Urban 


Rural 


Suburban 


Practice computations 


5.06 


4.35 


5.53 


Practice word problems 


4.24 


4.16 


4.53 


Problems solved more than one way 


4.26 


4.49 


4.68 


Problems that require you to really think 


4.10 


4.87 


4.62 


Problems where you explain your thinking 


3.47 


3.46 


3.39 


Problems that take at least a week to complete 


2.22 


2.45 


1.63 


Problems that apply to real life 


3.46 


3.58 


3.61 


Use calculators to solve problems 


4.79 


4.44 


4.91 


Use rulers, blocks, or solids 


3.38 


3.18 


2.97 


Give an oral presentation 


2.18 


2.30 


1.62 



Note. 1 = Hardly at all, 6 = A couple of times a week or more, 
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they practice computations: Students in all classroom types, particularly those 
in suburban classrooms, reported frequent engagement in such practice. 
Computation practice, in fact, in general was. the highest frequency activity of 
all those queried. No significant differences were found across school type for 
any of these activities. 

Tables 30 and 31 display teachers' and students' responses regarding the 
frequency of another activity associated with innovative instructional practice: 
working in small groups. Students were consistently more conservative in 
their frequency estimates than were teachers, but according to the reports of 
both teachers (x 2 (10) = 24.77, p = .006) and students (F(2) = 3.64, p = .04), it is 
clear that students in suburban classes were unlikely to be engaged regularly 
in small-group work, while rural students in our sample were most likely to be 
so engaged. 

Table 30 

Percentage of Teachers Who Reported Work in 
Small Groups at Least Once a Week by School 
Type (Teacher Interview Results) 



Type of school 


Teachers* 


Urban 


63.6 


Rural 


83.3 


Suburban 


0.0 


*p < .05. 




Table 31 




Mean Comparisons of Student Frequency 


Ratings of Small-Group Work by School Type 


(Student Survey Results) 




Type of school 


Students* 


Urban 


3.41 


Rural 


4.43 


Suburban 


2.23 


Note. 1 = Hardly at all, 6 


s A couple of times a 


week or more. 




*p < .05. 
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We also asked students and teachers how often they worked on 
assignments that required extended writing (in the query to students, 
problems which required them to write a paragraph or more). Although 
student survey differences were not significant, teachers in urban and rural 
schools were more likely than teachers in suburban schools to report such 
activity (x 2 (10) = 19.39, p = .04) (see Tables 32 and 33). 

Composite opportunity to learn scales. Based on a combination of factor 
analysis and theoretical assumptions, students' responses regarding specific 
classroom practices were combined into three overall scales. These provide an 
more reliable test of differences in students' opportunity to learn. Table 34 
displays the scales and the individual items which constitute them; also 

Table 32 

Percentage of Teachers Who Reported Working 
on Problems Requiring Writing at Least Once 
a Week by School Type (Teacher Interview 
Results) 



Type of school 


Teachers* 


Urban 


73.9 


Rural 


83.3 


Suburban 


66.7 


*p < .05. 




Table 33 




Mean Comparisons of Student Frequency 


Ratings of Working on 


Problems Requiring 


Writing by School Type (Student Survey 


Results) 




Type of school 


Students 


Urban 


2.94 


Rural 


2.75 


Suburban 


2.36 



Note. 1 = Hardly at ali t 6 = A couple of times a 
week or more. 
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Table 34 

Classroom Learning Opportunity Scales: Composite 
Items and Reliabilities (Student Survey Results) 

Mathematical communication scale (alpha=.73) 

Problems that require a written paragraph 

Problems which require you to explain your thinking 

Problems that require you to really think 

Problems that can be solved in more than one way 

Give an oral presentation 
Applied problem solving scale (alpha=.66) 

Use rulers, blocks or solids 

Practice word problerns 

Work in small groups 

Problems that apply to real life 
Topic preparation scale (alpha=.76) 

Perimeter 

Graphing data 

Distance/time 

Fractions 

Ratios 



displayed are the reliabilities (measures of how the scales hold together) 
associated with each scale, based on Cronbach's alphas (Cronbach, 1951). 

The "communication" scale is made up of items that indicate how often 
students practice problems that require them to communicate how they are 
thinking. The "applied" scale refers to how often students practice practical 
problems using applied methods or real-life perspectives. The "preparation" 
scale focuses on how well prepared students felt for the specific math 
concepts required on the CLAS common form. Nonsignificant trends in 
Table 35 suggest that students in urban and rural schools (the rural students 
more so than the urban students) spend more time than students in suburban 
schools on applied problem solving. However, students in suburban schools 
appear to feel better prepared than urban and rural students for selected 
mathematics concepts required on the common form assessment; and 
suburban students also practice computations more regularly than other 
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Table 35 

Opportunity-to-Learn Composite Scales: ANOVA Findings by School Type (Student 
Survey Results) 





Communication 


Applied 


Preparation 


Computation 


Urban 


16.90 


14.49 


13.87 


5.06 


Rural 


17.87 


15.31 


13.87 


4.35 


Suburban 


16.78 


13.39 


15.63 


5.53 



students in our sample. ("Computation* denotes frequency of practice in 
computation, a single item unrelated to other subscales.) 

Homework. Students' responses about the frequency with which they 
were assigned homework and the difficulty level of that homework provide a 
possible window into why suburban students tend to report themselves better 
prepared than other students in our sample (see Table 36). Time on homework 
presumably represents learning time and thus additional opportunity to learn. 
In this regard, suburban students reported being assigned math homework 
more often than did urban students, who in turn reported more homework 
than did rural students (F(2) = 3.67, p = .04). Whereas suburban students 
reported having homework four to five nights a week on average and urban 
students reported having homework about three nights a week on average, 



Table 36 

Mean Comparisons of Student Ratings Regarding Homework by School Type 
(Student Survey Results) 



Type of school 


How often homework 
is assigned 3 


How long it takes to 
finish homework^ 


How difficult 
homework is c 


Urban 


5.08* 


2.47 


2.98 


Rural 


3.62* 


2.65 


3.05 


Suburban 


6.50* 


2.24 


3.16 



a 1 = Never, 7 = Every night. 
b 1 * 15 minutes, 5 = More than one hour. 
c 1 a Very easy, 5 = Very difficult. 
*p < .05. 
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rural students reported homework assignments only once or twice per week on 
average. No differences were found the time students report spending on their 
homework assignments (30 to 45 minutes on average) or in the difficulty level 
of that homework ("moderate" on average). 

Access to learning assistance outside of the classroom. We asked 
students to report the amount of help they received outside of class from 
teachers, friends, and family. There were no significant differences across 
school types in the amount of help students reported receiving from their 
family, with students reporting on average that they "sometimes" get help 
from family members. Likewise, no differences were found across school type 
in help received from teachers, with students reporting on average receiving 
help "sometimes" to "usually" from their teachers. Differences did emerge, 
however, in patterns of help students reported getting from friends. Rural 
students reported on average "sometimes" or "usually" getting help from 
friends, whereas urban and suburban students reported "sometimes" at best 
(rural moan was 2.43; urban and suburban means were 1.92 and 1.90, 
respectively; F(2) = 5.23, p= .01). 

Preparation for the CLAS 

Students' perceptions of preparedness. Teachers and students also were 
queried about their direct preparation for the CLAS. Table 37 shows how well 
students felt they were prepared for the CLAS. A one-way analysis of variance 
indicated that suburban students are significantly more confident about their 
preparedness for the CLAS than other students (F(2) = 4.15, p = .03). It possible 
that— having done well on previous standardized tests— students in suburban 
schools generally have more academic self-confidence than students in either 
rural or urban schools. 

Teacher reports on direct preparation. Almost all teachers indicated that 
they engaged their students in specific activities to prepare students for the 
CLAS. The state provided schools with a "CLAS Mathematics Sampler" to 
acquaint teachers and students with the type and nature of assessment they 
would encounter on the CLAS. The Sampler, as the name implies, included 
sample problems, and teachers were free to assign and work through these 
problems with their students. The great majority of teachers interviewed 
(91%) had both seen ths Sampler and used it to prepare their students for the 
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Table 37 

Mean Comparisons of Student Ratings of Their 
Preparation for the CLAS by School Type 
(Student Survey Results) 



Type of school 


Preparation for the CLAS* 


Urban 


2.73 


Rural 


2.40 


Suburban 


3.21 



Note. 1 * Not at all, 4 ■ Very much so. 
*p < .05. 



assessment, although rural teachers appeared less likely than other teachers 
to have done so (see Table 38). On average, teachers reported devoting between 
3 and 8 periods tc practice with the Sampler. 

How well teachers expect their students to do on the CLAS. Teachers 
were asked to estimate the percentage of their students they expected to do well 
on the CLAS open-ended and multiple-choice items. For the most part, 
teachers tended to think that less than half of their students would do well on 
the open-ended portion of the assessment, and that a higher proportion would 
do well on the multiple-choice items. As Table 39 shows, suburban teachers 
held the highest expectations for their students' performance on the multiple- 
choice items (F(2) = 4.85, p = .014). On average, suburban teachers expected 
about 75% of their students to do well on the multiple-choice portion of the 
assessment. No significant differences were found by school type in teachers' 
expectations of their students' performance on open-ended items. 

Table 38 

Percentage of Teachers Using the CLAS 
Sampler by School Type (Teacher Interview 
Results) 



Type of school 


Used CLAS Sampler 


Urban 


95,7 


Rural 


66.7 


Suburban 


100.0 
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Table 39 

Mean Comparisons of Teachers' Expectations of How 
Many of Their Students Will Perform Well on the 
CLAS by School Type (Teacher Interview Results) 



Type of school 


Open-ended 


Multiple-choice* 


Urban 


3.87 


4.09 


Rural 


3.83 


3.67 


Suburban 


4.17 


5.00 


Note. 1 = None, 


6 = Almost all. 





*p < .05. 

Summary 

Our analyses show both similarities and differences across school types in 
students' opportunity to learn and their direct preparation for CLAS: 

With regard to resources, results indicate that the rural teachers in our 
sample tended to have less background in mathematics than other teachers, 
both in terms of undergraduate major and in terms of participation in recent 
professional development activities. In the area of instructional resources, 
urban schools had relatively less access to recent mathematics texts. No 
differences were found in terms of availability of calculators, although urban 
schools tended to provide them for their students while suburban students 
tended to bring them from home. 

In terms of instructional practices, classrooms in all schools tend to place 
the most emphasis on computational problems, followed by word problems, 
problems that require thinking or that can be solved in more than one way, 
and the use of calculators. Less emphasis is placed on problems that require 
an explanation, problems that take at least a week to complete, real-life 
problems, and the use of rulers, blocks or solids. Rural and urban students 
were more likely to be engaged in two other classroom practices that are 
associated with the development of complex thinking and problem solving: 
cooperative group activity and extended writing. In contrast, suburban 
students were more likely to have nightly homework. Finally, it is the teachers 
in suburban schools who are more confident than other teachers of the match 
between their teaching practices and the CLAS. 
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Almost all teachers reporting spending at least a number of class periods 
preparing their students for the CLAS. Mirroring teachers' views about the 
match between their instructional programs and the CLAS as well as 
teachers' expectations for their students' performance on the assessment, 
suburban students in our sample felt better prepared than their peers for 
specific concepts that were present on the CLAS common form and more 
confident of their preparedness for the CLAS in general. 

SUMMARY, CONCLUSION, AND NEXT STEPS 

The preliminary findings presented in this report suggest that alternative 
assessment, as represented by the open-ended tasks included on the CLAS 
assessment, is achieving at least some of its aims. Students seem to 
understand the differences in approach necessitated by open-ended and 
multiple-choice problems. In addition, because they have to explain their 
thinking and communicate their knowledge in a variety of forms, students 
find the former type of problem both more interesting and more challenging 
than traditional forms of assessment. By not giving students alternatives, 
furthermore, open-ended items clearly inhibit simple guessing strategies and 
instead encourage students to pursue a mathematics-based reasoning 
approach to solve problems. This is not to say, however, that students "like" 
open-epded items more than multiple-choice ones. In fact, they express 
preference for the latter, perhaps because students find comfort in the 
familiarity of multiple-choice items, think they will perform better on the these 
items, and better understand how their performance on these items will be 
assessed. 

A majority of teachers in the sample already engage their students at 
least weekly in many of the instructional activities that the CLAS is intended to 
encourage: word problems, problems that can be solved in more than one way, 
problems that require extended writing, use of calculators, problems that 
require students to really think, and small-group work. However, students 
perceive that other types of activities associated with a thinking curriculum 
are less prevalent: Problems in which students explain their thinking, oral 
representations, projects that take a week or more to complete, use of 
manipulatives, and real-life problems are less visible in the curriculum. As 
an additional indicator of their routine practice, teachers clearly expect their 
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students to do less well on innovative open-ended items than on traditional 
multiple-choice ones. 

Given the equity impetus to our inquiry, it is encouraging that the urban 
students in our sample were not limited to a meager "drill and kill" 
curriculum. Nonetheless, that students in urban classrooms were more likely 
to have questions about a key concept in mathematical thinking, "assumption," 
and the fact that they had less access to recent texts raise questions about their 
preparation. Suburban students clearly feel better prepared for the 
assessment. 

We end by reiterating that the results reported here are preliminary. We 
are in the process of integrating them with other available data from the state, 
including actual student performance on the CLAS and individual student 
demographic data. Thes. -ill enable us to look more closely at the 
interrelationships among an etween student demographics, instructional 
practices, attitudes, and performance. Also of interest will be more detailed 
analyses of potential differences in opportunity to learn according to specific 
class enrollments— basic eighth-grade math, pre-algebra, algebra— as well as 
relationships between students' performance on alternative assessment as 
compared to other, more traditional indicators of their performance. 
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Classroom Observation 
CLAS Middle Grades Performance Assessment 
Special Grade 8 Math Study 



School: 



Date: 



Teacher: 



Period: 



Math Course: 



Time Start: 



A . Administration Context 

1. Administration Observed: 

□ State-level administration (DAY 1) □ Student-level pilot (DAY 2) 

2. Student grouping: 

□ Regular math classroom □ Special grouping: Describe 

3. Who administered the assessment? 

□ Regular classroom teacher □ Other adult: Who? 

4. Number of students taking this assessment: 

B. Assessment Presentation 

1. Did the administrator follow directions? □ Yes □ No 
If no, note deviation: 

2. How many students had questions? 



Tally: 
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3. What were the questions about? 

t 

4. Did the session include labeling student forms? □ Yes □ No 

5. What time did students start the assessment itself? _ 

. During the Assessment Period 

1. Do all students have access to calculators? □ Yes □ No 

2. How many calculators are available? 

What is the calculator-to-student ratio (1 calculator for every _ [number] 
students)? 

3. How are calculators made available (e.g., on each student's desk, shared, 
one per table, in back of room)? 

4 Were there any operational problems with the calculators (e.g., batteries, 
malfunction, etc.)? □ None observed □ A few □ Many 

5. How many students used calculators? 

Tally: 

Over entire period, what percentage of students used calculators? 

□ Almost all □ About 50% □ A few 

□ About 75% □ About 25% □ None 



6. For what part of the assessment did students use the calculators? 

□ Mostly open-ended □ Mostly multiple-choice □ About half and half 



7. What questions, if any, did students have during the assessment? 



8. How did the administrator respond to student questions? 



9. How engaged/interested were students in working the open-ended 
questions? 

(a) Percentage of students appearing very interested: 

□ Almost all □ About 50% □ A few 

□ About 75% □ About 25% □ None 



(b) Percentage of students appearing neutral: 

□ Almost all □ About 50% □ A few 

□ About 75% □ About 25% □ None 



(c) Percentage of students appearing uninterested: 

□ Almost all □ About 50% □ A few 

□ About 75% □ About 25% □ None 



10. What percentage of students used most of the 15 minutes (or more) to 
answer the first open-ended question? 

□ Almost all □ About 50% □ A few 

□ About 75% □ About 25% □ None 



11. What percentage of students finished during the regular assessment 
period? 

□ Almost all □ About 50% □ A few 

□ About 75% □ About 25% □ None 



12. What time did the assessment period end? 

Length of assessment period (not including directions): minutes 

13. Were students who didn't finish given a comfortable option for completing 
the assessment? 

□ Yes □ Somewhat □ No □ Not needed 

Explain: 



D. After the Assessment Period 

1 What comments did the administrator make after the assessment? 
(Include both formal comments made to the entire class and intormal 
comments made to individual students.) 



2. What comments did students make after the assessment? (Include both 
student-to-administrator comments and student-to-student comments.) 
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Student Survey 
CLAS Middle Grades Performance Assessment 
Special Grade 8 Math Study 



Directions : This is a questionnaire about the CLAS Middle Grades Performance Assessment you 
took. We want to know about the types of things youVe done in your math class that may have helped 
you on the assessment. We also want to know your reactions to the assessment. There are no right or 
wrong answers. Just respond as honestly as you can. Please read the following questions with me 
and mark or write your answers in the spaces provided. Your answers will be confidential. 



School: 

Math Teacher: 
Math Course: 



Your Name: 

Date: 

Period: 



1 ♦ In your current math cJaaft how often do you: 

(Circle one number on each line) 

A couple 

Once or of times 

Hardly twice per Every two a week 

at all semester Monthly weeks Weekly or more 



a. Practice doing 
computations? 

b. Practice doing word 
problems? 

c . Practice doing problems 
that can be solved in more 
than one way? 

d. Practice doing problems 
you have to really think 
about to come up with a 
solution? 

e. Practice doing problems 
that ask you to explain (in 
writing) your thinking? 

f . Do math projects that take 
at least a week to complete? 

g . Apply math to real-life, 
practical problems? 

O 
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1 2 3 

1 2 3 

1 2 3 

1 2 3 

1 2 3 

1 2 3 

1 2 3 



4 5 6 

4 5 6 

4 5 6 

4 5 6 

4 5 6 

4 5 6 

4 5 6 



In your current math class, how often do you: 

A couple 

Once or of times 

Hardly twice per Every two a week 

at all semester Monthly weeks Weekly or more 



h . Work in small groups to 
answer problems? 

i , Use calculators to solve 
problems? 

j . Use rulers, blocks, or solids 
to explore problems? 

k . Give an oral presentation? 



1 2 3 4 5 6 



I . Answer math problems 1 
that require you to write a 
paragraph or more? 



2 3 4 5 6 



2. How well have your math classes prepared you to answer questions in the following areas? 

(Circle one answer on each line) 

Little Very Don't 

or None Somewhat well know 



a. 


Fractions 


L 2 


3 


DK 


b. 


Area 1 


L 2 


3 


DK 


c . 


Perimeter 


L 2 


3 


DK 


d. 


Graphing data 


L 2 


3 


DK 


e. 


Distance/Time problems 


1 2 


3 


DK 


f. 


Using ratios 


1 2 


3 


DK 



3 . How often do you have to do homework for your math class? 

q Every night q Twice a week q Only when I don't finish my classwork 

q Four nights a week q Once a week q Never 

q Three nights a week 

4. How long does it usually take you to finish a day's math homework? 

q 15 minutes q 45 minutes q More than one hour 

q A half hour q One hour 
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5 . How would you describe the difficulty level of your math homework? 

q Very easy q Pretty easy q So-so q Pretty difficult q Very difficult 



6. When you have problems with your homework, how often do 

a . someone in your family? q Almost never 

q Sometimes 

b. a friend? q Almost never 

q Sometimes 

c . a teacher? q Almost never 

q Sometimes 



you get help from: 

q Usually 

q Almost always 

q Usually 

q Almost always 

q Usually 

q Almost always 



7. Do you have access to a calculator at home? q Yes q No 

If so, what kind? q Simple calculator q Scientific calculator 



8. How much do you agree with the following statements? 

(Circle one number on each line) 

Not at Moderately Very much 

all Somewhat so so 



a . I like math. 

b. I am good in math. 

c. Learning math is mostly memorizing. 

d . I like to do computation problems. 

e . I like to do math problems that don't 
have an obvious solution. 

f . I'm frustrated by problems that have 
more than one correct answer. 

g . I like to do problems that take a lot of 
time to solve. 

h. I like to work on projects that require 
me to use math. 

i . I think math will be useful to me in the 
future. 

j . I was well-prepared to take the CLAS 
Middle Grades Performance 
Assessment. 



2 
2 
2 
2 
2 
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3 
3 
3 
3 
3 



3 
3 



4 
4 
4 
4 
4 



9. The CLAS Middle Grades Performance Assessment you took had open-ended questions, v 
which you had to explain your answers, and multiple-choice questions- Compare how you 
about these two types of questions in the following areas: 

(Circle one number on each line) 

Open- Multiple- About the 
ended choice same 



a . On which type of question did you £rx harder? 

b. Which type of question did you find most 
interesting? 

c . On which type of question do you think you 
did better? 

d . Which type of question do you think showed 
better what you know about math? 

e . Which type of question did you like better? 



Thank You for Helping Us! 
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Student Interview 

Think Aloud Description of Processes Engaged in Solving Actual CLAS Problems 

and Reactions 

CLAS Middle Grades Performance Assessment 

Special Grade 8 Math Study 



School: 

Course: 

Student Name: _ 
Student Number: 



Math Teacher: 

Period: 

Date: 



I ntroduction/Directions : We're interested in learning what eighth graders think 
about when they do math problems on the CLAS Middle Grades Performance 
Assessment. We also want to know your reactions to the problems — how you feel 
about them compared to multiple-choice questions and how well your math 
classes have prepared you to do well on this assessment. There are no right or 
wrong answers so just answer as honestly as you can. 

I'm going to ask you to try to remember what you were thinking about as you went 
about answering some of the problems on the CLAS Middle Grades Performance 
Assessment and then ask you for your reactions. I want you to tell me everything 
that comes to your mind as you review your work. Fm going to be recording our 
talk today, okay? All your responses will be confidential. We will not be sharing 
them with anyone at school. 



1. Open-ended question number: □ 1 □ 2 

A. Using the student's assessment booklet, read the CLAS question with the 
student, then ask the student: 

(1) What do you think you're being asked to do? (What's being tested here? 
What are you supposed to do?) 



(2) What do you think the teachers will be looking for when they read your 
answer? (How do you think this will be graded?) 
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B. Please share with me how you went about answering the question: 

(1) What did you do first? What were you thinking about? 

(2) What did you do next? What were you thinking about? 

(3) What did you do next? What were you thinking about? 

(4) What did you do next? What were you thinking about? 



C. How well do you think your math class this year prepared you to answer 
this question? 

□ Very well □ Some □ Hardly at all □ Don't know 

Specifically why do you think so? (Prompt if necessary to tie to what 
student mentioned in B above.) 



GO 



D, How often in your math class have you done: 

(1) Problems that don't have an obvious solution? 

□ A couple of times a week or more 

□ Weekly 

□ Monthly 

□ Once or twice per semester 

□ Hardly at all 

(2) Problems that ask that you explain your thinking? 

□ A couple of times a week or more 

□ Weekly 

□ Monthly 

□ Once or twice per semester 

□ Hardly at all 



2, Multiple-choice question 1: 

A, Using the student's assessment booklet, read the CLAS question with the 
student, then ask the student: 

(1) What do you think you're being asked to do? (What's being tested here? 
What are you supposed to do?) 



(2) What do you think the teachers will be looking for when they read your 
answer? (How do you think this will be graded?) 
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B. Please share with me how you went about answering the question: 
(1) What did you do first? What were you thinking about? 



(2) What did you do next? What were you thinking about? 



(3) What did you do next? What were you thinking about? 



(4) What did you do next? What were you thinking about? 
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C . How well do you think your math class this year prepared you to answer 
this problem? 

□ Very well □ Some □ Hardly at all □ Don't know 

Specifically why do you think so? (Prompt if necessary to tie to what 
student mentioned in B above.) 



3. Think about the two open-ended questions you answered and the multiple- 
choice questions you answered and then answer the following questions: 

A. Which kind of question makes you think harder or is more challenging? 
□ Multiple-choice □ Open-ended □ No difference 



B. Which kind of question best lets you show what you know about math? 



Why do you think so? 



C. Which kind of question was easier for you to understand what to do? 



Why? 



□ Multiple-choice 



□ Open-ended 



□ No difference 



□ Multiple-choice 



□ Open-ended 



□ No difference 



Why do you think so? 



D, Which kind of question do you like better? 
□ Multiple-choice □ Open-ended 



□ No difference 



Why? 
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E. What do you like best about open-ended questions? 



F. What do you like least about open-ended questions? 



When you answer open-ended questions, do you think any differently or do 
anything differently than when you answer multiple-choice questions? 

□ Yes □ No If yes, please explain. 



H. Do you guess on tests? □ Yes □ No 

If so, on which kind of question— multiple-choice or open-ended — do you 
guess more? □ Multiple-choice □ Open-ended 

Why? 



I . What would be your advice to another student who has to prepare for an 
assessment like this? 



THANKS FOR YOUR HELP1 
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Teacher Interview 
CLAS Middle Grades Performance Assessment 
Special Grade 8 Math Study 



School: Date: _ 

Teacher: Period: 

Math Course: 



Gender: □ Male □ Female 

Ethnic Background: □ African American 

□ American Indian or Alaskan Native 

□ Asian or Pacific Islander 

□ Hispanic 

□ White - not of Hispanic origin 



A. Your Background and Experience 

1. Your college background: 

Undergraduate major: 

Undergraduate minor, if any: 



2. The type of teaching credential you currently hold: 

Q Certified in Math 

□ Certified in Middle School/Junior High Teaching 

□ Certified in General Elementary 

□ Provisional or Emergency in Math 
a ESL/Bilingual 

□ Other (specify): 



3. Your teaching experience, ^minting this year: 

a. Years of teaching: years 

b. Years teaching math in grades 7-12: years 

c. Years at this school: years 



4 Please indicate the number of quarter or semester credit hours that you have 
accumulated at the undergraduate and graduate levels in math or math 
education. 

Number of semester or trimester credit hours: 

Number of quarter credit hours: . 



5 During the last 3 years , what is the total amount of time you have spent on in- 
service education in n*tbp™ a tirs or the teaching of mathematics ? (Include 
attendance at professional meetings, workshops, and conferences, but do net 
include formal courses for which you received college credit.) 

Q N one □ 6 to 15 hours □ More than 35 hours 

□ Less than 6 hours □ 16 to 35 hours 



B. Your Curriculum and Instruction This Year 

T n pnswpring thpss qnpstinns. dI p asp focus on the class v.'s observed . 

6. What was the primary textbook you used in this class? 

a. Title 



b. Author 



c. Publisher 



d. Date of Publication 
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7. What chapters have you covered? 

Chapters: 

8. Do you use any other printed materials, such as an additional text or 
supplemental readings, with this class? 

□ Yes □ No 

If yes: Title Type of Material* 

a. 

b. ; 

c. . 

d. 

e. : 

f . 

*(l=textbook; 2=commercially prepared curriculum materials; 3=article 
from math or science publication; 4=locally developed materials; 5=other) 



9. Describe the major assignments (projects, reports, etc.) youVe given students this 
quarter. 



6? 
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10. How much class time did you spend on each of the following topics in your class 
this year? 



a. Fractions 

b. Area 

c. Perimeter 

d. Graphing data 

e. Distance/time problems 

f. Proportional reasoning 



Not 
covered 



1-2 
class 
periods 



3-5 
class 
periods 



6-10 
class 
periods 



2 
2 
2 
2 
2 
2 



3 
3 
3 
3 
3 
3 



4 
4 
4 
4 
4 
4 



Move 
than 10 
periods 



5 
5 
5 
5 
5 
5 



11. How often do you ask your students to reflect upon and explain their 
mathematical reasoning in the following situations? 

Once or Once or A couple 

twice a twice a times 

Never quarter Monthly month Weekly a week 



a. In small groups 

b. In written 
assignments 

c . As part of class 
discussions 



12 3 
12 3 

12 3 



4 5 6 

4 5 6 

4 5 6 



12. On the quizzes, tests, and big exams that you give to your students, about what 
percent of the items are: 

Quizzes/Minor tests Big exams 



a. multiple-choice questions? % % 

b. short-answer questions? % % 

c . open-ended problems? % % 

d. extended investigations taking a whole % % 

period or more? 

e. other? (specify) % % 



13. How often are students asked to solve open-ended items similar to those on the 
CLAS Middle Grades Performance Assessment (i.e., items where there are 
no obvious solutions and no single correct strategy, and that ask students to 
explain their reasoning)? 

□ Never □ Monthly □ Weekly 

□ Once or twice a quarter □ Once or twice a month □ A couple of 

times a week 



14. Do your students keep math portfolios? □ Yes □ No 



15. Describe how calculators or computers are used in your mathematics 
instruction, if at all. 



If not used, why not? 



a. How many of your students have calculators? 

(1) How many are provided by the school for use in the classroom? 

(2) How many are brought from home? 



b. How has the school prepared your students to use calculators? 



What percent of your class is competent in using a calculator? 



% 



d. Do you permit unrestricted use of calculators? □ Yes □ No 
If no, do you permit students to use calculators on tests? 
□ Yes □ No 



C. Reactions to CLAS 

16. Have you seen the Mathematics Sampler or Addendum describing the new CLAS 
Middle Grades Performance Assessment? 

□ Yes □ No 

17. Compare your classroom curriculum and instruction to what you expected to be 
on the assessment: 

a. To what extent is your classroom instruction (including texts, your teaching, 
and assignments) aligned with the new CLAS-type assessments? 

□ Excellent match □ Some match □ Not sure 

□ OK match □ Poor match 

(1) How are they similar? 
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(2) How are they different? 

b. What percent of your students do you expect to do well on the open-ended 
items? 

□ Almost all □ About 50% □ A few 

□ About 75% □ About 25% □ None 

Please explain the reason for your rating: 



c. What percent of your students do you expect to do well on the multiple-choice 
items? 

□ Almost all □ About 50% □ A few 

□ About 75% □ About 25% □ None 

Please explain the reason for your rating: 

d. Did you use exercises from the Mathematics Sampler or Addendum to 
prepare your students? 

□ Yes □ No 

If yes, about how many class periods did you spend on them? periods 

e. How did you otherwise prepare your class for the CLAS Middle Grades 
Performance Assessment? 

i. Attitude: 



ii. Content: 
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111. 



Procedures: 



iv. Assessment-taking strategies: 



18. What are your overall reactions to the new CLAS Middle Grades Performance 
Assessment? 



19. How well have you been prepared to teach to CLAS-type objectives/ 
□ Very well □ So-so □ Not well 

Explain: 



20. Have any of your students been excluded from taking this assessment? 
□ Yes □ No 

If yes, how many? 

why? 



7 o 

8 



i 
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D. Your Students 

21. How are students assigned to this class? 

□ Teacher or counselor recommendation 

□ Student elective 

□ Placement or achievement tests 

□ Required 

□ Other 

If placed, can students waiver in? □ Yes □ No 

22. How would you rate this class' mathematics achievement level overall? 



□ Top 10% 

□ Upper quartile 



□ Above average 

□ Average 



□ Below average 

□ Lower quartile 



Compared to other students in this school? 

□ Top 10% □ Above average 

□ Upper quartile □ Average 



□ Below average 

□ Lower quartile 



Compared to other students in this district? 

□ Top 10% □ Above average 

□ Upper quartile □ Average 



□ Below average 

□ Lower quartile 



Compared to other students in the state? 

□ Top 10% □ Above average 

□ Upper quartile □ Average 



□ Below average 

□ Lower quartile 



23. Is there anything else you'd like to share about your professional opinions or 
reactions to the CLAS Middle Grades Performance Assessment? 
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To speed up the processing of your honorarium, please provide your social security 
number and home address in the space below. 

SSN: 

Name: 

Home address: 



THANK YOU FOR YOUR TIME AND THOUGHTFULNESS! * 



* Remind teacher about collection of: (a) a description of the major assignments given to students in 
class this year (projects, reports, etc.); (b) samples of tests and quizzes given during Spring semester; 
(c) a copy of the final exam for the class (if available). 
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